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INTROOUCTION 



Various personnel decisions are based on prediction of success in 
a given task. To the degree that this prediction can be achieved, the 
accuracy of personnel selection, classification, and placement, and the 
design of training programs would improve. Unfortunately, the procedures 
required to insure a satisfactory level of prediction are arduous and time- 
consuming. These procedures consist of the following necessary stages: 
1) task or job analysis, to form a hypothesis as to what characteristics 
make for success or failure; 2) choice of possibly useful tests to measure 
these characteristics; 3) administration of tests to an experimental group 
of workers; 4) collection of criterion data that show how these workers 
succeeded on the job; 5} analysis of the relation between test score and 
success on the job, and Installation of the most effective selection plan; 
and 6) periodic validity studies to check on the continuing soundness of 
the plan (Cronbach, 1970, p. 407). 

Of these stages, one of the most crucial and certainly one of the most 
difficult to conduct is task or job analysis. The difficulty inherent in 
task analysis is apparent when one reviews the trar^ndous variety of methods 
and the sheer number of different schemes proposed for accomplishing it 
(see Smith, 1965). In effect, task analysis consists of two distinct 
aspects: a description of the salient characteristics of the tasks, and 
the translation of these characteristics into hypotheses stating the 
operator characteristics necessary for successful performance. In terms 

of the accuracy and eff1;1ency of such an analysis, it is desirable to have 
a common "language" to dj-scribe both the task and operator characteristics. 
Furthermore, it is important that the cofranon language consists of constructs 
having a high degree of reliability and validity. That is, the vocabulary 
should include only those terms which can be used consistently and which 
hive been enpirlcally demonstrated to be effective predictors of performance. 

In addition to being able to describe both the task and the operator, 
a powerful task-analytic language must also satisfy other requirements. 
F1r.,t, as implied by Cronbach 's list of stages, the constructs conprlsing 
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the vocabulary must be operationally defined, typically through a reliable 
test or set of tests. Practically, these tests must be amenable to admin- 
istration to groups of operators, have easily interpretable scores, etc. 
Second, the constructs or their defining tests must predict performance 
in a selected criterion task. Minimally, this prediction must differentiate 
good operators from poor operators. Other predictions and differentiations 
are also desirable; e. g., predicting who will be good learners and poor 
learners of the criterion task, or who will better retain the training. 
Third, the constructs must be valid and reliable over time. For example, 
a test which differentiates good and bad operators might not be useful 
if, after some experience on the task, the test becomes irrelevant to 
performance. 

In order to address these requirements, FleisNnan and his co-workers 
have focused on the developnent of taxonomies for the description and classi- 
fication of tasks (e.g., Fleishman, Kinkade, & Chambers, 1968; FleisNnan 
& Stephenson, 1970; FleisNnan, Teichner, & Stephenson, 1970; Theologus, 
Romashko, & FleisNnan, 1970; Wheaton, Mirabella, & Farina, 1971; Levine, 
Romashko, & FleisNnan, 1971; Farina & Wheaton, 1971; Wheaton & Mirabella, 
1972; and Wheaton, Shaffer (Eisner), Mirabel la, & FleisNnan, 1973). One 
of the taxonomic languages investigated by these researchers for the 
description of tasks and operators uses anpirically defined human abilities 
as its vocabulary. In this context, ability refers to a characteristic 
trait of an individual which has been inferred frm behavioral consistencies 
(e.g., correlations) on several kinds of tasks. It has been found that 
this ability language meets several of the requiranents discussed above. 
Since the abilitie.. are empirically defined (through factor-analytic pro- 
cedures) by response consistencies on tests tf known reliability, operational 
definitions of abilities are available. The application of an abilities 
language to the classification of tasks has been moderately successful. 
Given verbal definitions of abilities derived from empirical firKlings, 
Theologus, Romashko, & FleisPman (1970), Theologus & FleisNnan (1971), and 
Levine, Greenbaum (Kramer), & Notkin (1973) demonstrated that a set of 
observers could consistently select a defining set of abilities for several 
different tasks. 



However, it soon became apparent that a different set of considerations 
was involved when attempting to predict level of criterion task performance, 
as opposed to merely describing tasks in terms of ability re<?uirements. 
One consideration is that in order to make a prediction, one must discover 
not only which abilities are involved in task performance, but also 
their relative levels of invclvanent. A second consideration is that for 
any task, either the descriptor abilities or their relative weights may 
vary from one specific configuration to another. For example, assione that 
It is possible to describe the task of a passive sonar operator in terms of 
an abilities language. Assume also that the specific abilities selected 
to describe this task may differ as a function of changes in the specific 
characteristics of the Usk. For example, the sonar signals might appear 
in a low-noise background, in which case they w)uld be easily discriminable, 
or a high-noise background in which case they would not be easily discn'm- 
inable. As the "dlscrlminabillty" characteristic of the task varies, there 
might be a corresponding change in the abilities required for satisfactory 
performance. "Pitch discrimination" would probably not be an important 
predictor ability in the low-noise, high dlscriminability case. However, 
it might be important if the signals were embedded in high-background noise 
and, therefore, were not easily distinguishable. Similarly, "memory" would 
not be important If only two different signals were presented, but might 
be crucial If several different signals were to be discriminated.* To 
state the problan in terms of a constraint on the methodological language. 
It Is important that: the language and its constructs be sensitive to 
changes in task characteristics when in fact these changes alter the 
demands paced on the operator and thus require different operator aptitudes. 
On the other hand, certain changes In task characteristics, while affecting 
level or variability of task performance, might not require different 
operator aptitudes. A powerful task-analytic language should be sensitive 
to f;1ther of these relationships. 

Thus, there is a need for systematic investigation of the types and 



*Wheaton, Shaffer (Eisner), Mirabella, & Fleislwian (1973) actually 
studied hypothetical sonar tasks in this way. Their report (first in this 
series) describes their actual findings. 



magnitudes of changes 1n task characteristics, and the role each plays in 

dt'termining necessary operator abilities. 

"As new systems, jobs, and tasks are developed, forecasts are 
needed regarding the kinds of personnel who will be required 
in order to perform effectively at these positions. Similarly, 
as existing equipment and procedures are updated, estimates are 
needed of the expected impact of such modifications on per- 
formance (Wheaton, Shaffer (Eisner), Mirabel la, & FleisNnan, 
1973, p. 1)." 

The general goal of the current research is to develop principles 
relating task characteristics to ability requiranents. More specifically, 
this phase of the program addresses the following issues: 

1. Can an abilities vocabulary adequately relate operator and 
task characteristics by means of a camnon language? 

2. Do ability constructs predict performance in a range of different 
criterion tasks? 

3. Is the ability structure for a given task sensitive to changes 
In task performance denands? 

4. How does this "sensitivity" manifest itself? 

The strategy adopted to investigate these issues was discussed more 
fully in the previous report In this series (Wheaton, et al., 1973). In 
essence, Cronbach's stages of test establisNnent were applied to classes 
of tasks representative of the kinds of tasks prevalent In the modern Navy 
and of theoretical interest. Using this strategy, an early study investi- 
gated a psychomotor task (Fleishman, 1957); the current series initially 
focused on an auditory signal identification task representing the duties 
performed by a passive sonar operator (Wheaton, et al., 1973). Thus, 
previous research in this area has focused m< inly on less cognitive tasks. 
In order to determine whether or not the same proposed methodology would 
be useful in "higher-order" reasoning tasks, the current study examined 
an electronic fault-finding task, representative of situations faced 
by electronic troubleshooters. 

The troubleshooting task was carefully analyzed, and abilities 
hypothetically contributing to criterion performance were selected. 
Next, a series of manipulations of the basic fault-finding task was 
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introduced. These manipulations were assumed to affect the conditions of 
task performance by making the task more demanding while maintaining the 
basic performance requirements. These manipulations will be discussed 
further in the method section. The criterion task and the reference 
battery were then administered to a large group of subjects. To determine 
the relationship between task variations and ability requirements, several 
statistical analyses were then performed, including a factor analysis of 
the reference battery to establish an empirical ability structure, and an 
analysis to estimate the loadings of the various task criterion conditions 
on that structure. 

These procedures permitted a direct examination of several of the 
issues discussed above. First, the effect of changes in criterion task 
conditions on performance could be assessed. Second, the degree to 
which the set of ability factors predicts performance could be directly 
estimated. Third, the effect of the task variations on the efficacy of 
the predictions could be demonstrated. Finally, the nature of any 
systematic relationship between changes in task parameters and changes 
in the relevance of different abilities could be examined. 



METHOD 



Subjects 

The subjects employed In this study were 141 male college students 
with norm&l color vision recruited from universities in the metropolitan 
Washington, D. C. area. They were paid $18.00 for their participation 
upon completion of a single day of testing. Six subjects were dismissed 
when they failed to understand the experimental criterion task after 
receiving standard training. 

Experimental Criterion Task 

The criterion task consisted of a series of problems in which subjects 
were required to locate broken wires in hypothetical electronic devices. 
The basic format was a current-flow diagram or digital-logic circuit 
in which the state of the output at any point was determined by the pre- 
ceding logic gates. For example, in circuit I below, 

LIGHT 



1 








AND 


2 







o 




CIRCUIT I 



CIRCUIT II 



the light will not go on unless switches 1 and 2 are both depressed, 
allowing current to flow through the AND logic gate. In circuit II, the 
light will light if either switch 1 or switch 2 (or both) is depressed, 
since only one switch is necessary to permit current through the OR logic 
gate. Each fault-finding problem was constructed with a number of such 
AND and OR gates in the circuit. 

In each circuit a single faulty wire, or "breakpoint," was introduced. 
At such a point the current flow was disrupted. The subject's task was 
to identify the location of this break by probing the circuit at various 
locations while depressing different canbinations of switches. For 
example, consider circuit III below. 
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CIRCUIT III 



A and B represent the locations of potential breakpoints. If switch 1 is 
depressed and the light goes on, a break cannot exist at point A. If 
switch 2 is depressed and the light fails to go on, point B must be faulty. 
In the experimental problems {paper-and-pencil representations of the 
circuits), subjects had the opportunity to place a probe (a "light bulb") 
at various designated points ("sockets") In the circuit diagram in order 
to find which of the possible breakpoints was, in fact, faulty. Thus, 
each problem contained exactly one true breakpoint which the subject had 
to identify from among several potential breakpoints. In circuit III 
above, only one test was necessary to locate the actual breakpoint, since 
if A was not faulty, B must be, and vice versa. 

The troubleshooting task was varied in two ways. First, formal 
difficulty was manipulated by increasing the number of possible break- 
points and the number of gates in the circuit. The effect of this mani- 
pulation was to increase the "depth" of the necessary search. In the 
more conplex circuits it was necessary to trace back through as many 
as four gates in order to test a potential breakpoint. 

Three levels of formal difficulty were used. The first level was 
represented by problems with four gates and nine possible breakpoints; the 
second level contained five gates and sixteen possible breakpoints; and the 
third and most complex level had a total of six gates and twenty-four break- 
points. 

The second manipulation was to vary the perceptual complexity of the 
problem by changing the configuration or layout of the circuit. Circuit 
diagrams were created such that different circuits were topologically 
equivalent, but their spatial appearances were quite varied. This simulated 
the real-world situation where troubleshooters are often faced with 



perceptually confusing diagrams, in which it is neither imrodiately 
apparent how the circuit is organized, nor obvious how troubleshooting 
should proceed. 

Three levels of perceptual cqnplexity were designed. The first level 
of perceptual complexity was an uncomplicated left-to-right, switch-gate- 
light circuit diagram. In the second level, the locations of switches and 
gates remained the same but particular connections between switches and 
gates were interchanged so that proximal switches and gates were not 
necessarily interconnected, and wires from one switch crossed over wires 
from another. In the third and most ccmiplex level of perceptual complexity , 
positions of the switches and gates were changed, and the general left- 
to-right organization was disrupted. 

Nine different paper-and-pencil test problems were generated from 
this 3 X 3 ( formal difficulty by perceptual complexity ) design. Each 
problem was presented twice, with different breakpoints as solutions, 
providing a total of eighteen experimental problems. The nine test 
circuits presented to subjects are included in Appendix I. 

The order of presentation was counterbalanced by use of a Greco-Latin 
square, to control for practice effects. In addition, each problem used 
different letters to label light socket probe points, and different numbers 
to label the possible breakpoints. Three problens at the beginning and 
two problems at the end of the test session were added as buffer problems 
to insure that subjects understood the mechanics involved and to reduce any 
endspurt effect. 

Procedure 

Upon arriving, subjects filled out a brief biographical questionnaire. 
Next, subjects received training on the troubleshooting task. This con- 
sisted of Instruction on six sample circuits which illustrated the mechanics 
of probing for the fau'ity breakpoint; i.e., identifying the true break- 
point from among the potential breakpoints. This training period usually 
took 30 to 45 minutes. 
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Subjects were told that each problem In the testing session would 
have exactly c.e breakpoint and their task was to discover this breakpoint. 
Instructions emphasized that subjects would be scored in terms of speed 
(how fast they performed) and efficiency (number of breakpoints eliminated 
from consideration with each test of the circuit performed). A complete 
set of Instructions is presented in Appendix II. 

After training, twenty-three problems (18 test problems and 5 buffers) 
were administered to each subject. In order to locate the breakpoint each 
subject formulated a troubleshooting test, writing down on a small card 
the 5.witch (or switches) he wished to depress and the letter of the light 
he wished to test. This card was given to the experimenter who sat 
directly in front of the subject's desk. Feedback from the experimenter 
as to whether the light did o^^ did not light was provided through a 
visual display. This display consisted of a green "yes" light indicating 
that the light being tested would light, a red "no" light indicating the 
light would not light, and a yellow "error" light indicating the subject 
had performed a test for which there was no pathway connecting the switch 
and light, or a test in which the switches did not provide sufficient 
current through an AND gate to light a light. After feedback, the subject 
formulated his next test. 

The subject continued to write and sjbnit test cards, one at a time, 
until he thought he had located the breakpoint. He then wrote this 
breakpoint number on a card. If he was correct, the experimenter con- 
firmed his answer, and introduced the next problem. If the subject wrote 
down the wrong breakpoint, he was told to continue testing in order to 
determine the correct breakpoint. In this manner, the subject attempted 
to solve each of the 23 problems. Short rest breaks occurred after 
problems 3, 7, and 19. Subjects took one and a half to three hours to 
complete this phase. 

For each problem, a complete record of the subject's tests was made 
from the cards he submitted. The experimenter also recorded the time in 
seconds from the beginning of each problem to the first "negative test" 
(i.e., the first nonerroneous test on which the probe light failed to 



light), as well as the total time to solution of the problem.* If the sub 
ject had not solved a problem at the end of 15 minutes, he was told that 
his time limit was up and to proceed to the next problem. 

Reference Test Battery 

Staff members reviev^d definitions of many empirically determined 
abilities, and independently selected sets of abilities judged as relevant 
to the criterion task. Inter judge agreen«nt was quite high. The tests 
defining the selected abilities were then c(Mnbined to form a reference 
battery for administration to subjects. 

A battery of 21 tests was acbninistered to all subjects following 
their participation in the fault-finding task. The tests represented 
six well-established factors in the cognitive, perceptual, and memorial 
domains of performance. The specific factors chosen were hypothesized 
to be relevant to criterion task performance. To insure adequate factor 
definition, each of these factors was represented by a minimum of three 
tests. 

In assembling the battery, considerable use was made of the Kit of 
Reference Tests for Cognitive Factors prepared by French, Ekstrom, and 
Price (1963). While each test taken from the kit had two equivalent 
parts, only one part of each test was administered due to time limitations 
Three other tests from a set of tests developed by Rose (1974) were also 
included in the battery because of their judged relevance to criterion 
task performance. The entire test battery was composed of group tests 
of the paper-and-pencil variety. 

Brief descriptions of the reference tests are given below with 
references to further sources of Information. The reliability reported 
for each test Is shown in Table 1. In cases where this information was 
unavailable, reference Is made either to the original test from which the 
present version came or to a similar test. The order In which the tests 
were acbninistered is shown in the second column. 



* Data on time to the first negative test are not discussed in this 
report. They essentially parallel total time data. 
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TABLE 1 

RELIABILITIES OF REFERENCE TESTS 





Induction Factor 


Order 


r 


Source 


Letter Sets Test 
Locations Test 
Figure Classification 


5 
8 
21 


.64 
.82 
.94 


Lemke et al . (1967) 
Lfcmke et al . (1967) 
Pemberton (1952) 


Associative Memory Factor 








Picture-Number Test 

Object-Number Test 

First and Last Names Test 


20 
9 
7 


.76 
.79 
.81 


Duncanson (1966) 
Duncanson (1966) 
Duncanson (1966) 


Flexibility of Closure Factor 








Copying Test 
Closure Flexibility 

(Concealed Figures) 
Designs Test 


3 

15 
13 


2 

.88*^ 

.78 
.94 


Thurstone (1938) 

Buros (1965) 
Pemberton (1952) 


Perceptual Speed Factor 








Finding A's Test 
Number Comparison Test 
Identical Pictures Test 


10 
12 
6 


.81 
.79 
.88 


Duncanson (1966) 
Duncanson (1966) 
Duncanson (1966) 


Syllogistic Reasoning Factor 








Nonsense Syllogisms Test 
Logical Reasoning Test 
Inference Test 


14 
4 
17 


.88? 
.72*^ 
.53 


Lemke, et al . (1967) 
L«nke, et al . (1967) 
Guilford, et al. (1952) 


Spatial Scanning Factor 








Maze Tracing Speed Test 
Choosing a Path Test 
Map Planning Test 


2 

16 
18 


.94 
.77 
.79 


Frederiksen (1965) 
Frederiksen (1965) 
Frederiksen (1965) 


Other Reference Tests 








Graimtatical Reasoning 

(A-B) Task 
Neisser Search Task 
Permutations Task 


1 

11 
19 


.80 J 
.80j 
.83^ 


Baddeley (1968) 
Rose (1974) 
Rose (1974) 



All reliabilities, unless otherwise indicated, are split-half reliability 

2 coefficients corrected for full length with the Spearman-Brown formula. 

3 Reliability estimated by the tetrachoric correlation of odd and even items. 
- Kuder-Rlchardson 20 estimate. 

Pearson product moment test-retest reliability. 
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Reference Tests and Ability Factors . The Induction factor has been 
defined as the ability to find general concepts that will fit sets of 
data. It involves the formulation and testing out of hypotheses. The 
following three tests are marker tests for this factor: 

Letter Sets Test— Five sets of four letters each are 
presented. The task is to find the rule which relates four 
of the sets to each other and to mark the one set which does 
not fit the rule. There are 15 it©ns (seven mins.). Score 
is the (iiBuber correct minus a fraction of the number incorrect 
(French, et al . , 1963). 

Locations Test— Each problem consists of five rows of 
small dashes separated into groups of dashes by blank spaces. 
In each of the first four rows, one place in each row is marked 
according to a rule. The task is to discover the rule and to 
mark one of five numbered places in the fifth row accordingly. 
There are 14 problems in all (six mins.). Score is the number 
correct minus a fraction of the number incorrect (French, et al., 
1963). 

Figure Classification Test—Each item presents two or three 
reference groups, each containing three geometrical figures that 
are alike in accordance with sane rule. The second row of each 
item contains eight test figures. The task is to discover the 
rules and then to assign each test figure to one of the groups. 
There are 14 problems containing eight test figures (eight mins.). 
Score is the nianber correct minus a fraction of the number incorrect 
(French, et al., 1963). 

Associative Memory has been defined as the ability to remember bits 
of unrelated material. The marker tests are: 

Picture-Number Test— The subject studies pictures of common 
objects, each paired with a two-digit number. Later, when the 
pictures are presented to him In a different order, he is re- 
quired to write in the number associated with them. There are 
21 itsns in all (four mins. for m^norizing, three mins. for 
testing). Score is the number correct (French, et al., 1963). 

Object-Nianber Test— The subject studies 20 word-nimiber 
pairs arKJ must recall the appropriate nianber when the words 
are presented to him in a different order. There are 15 items 
(three mins. for menwrizing, two mins. for testing). Score 
is the nianber correct (French, et al., 1963). 

First and Last Names Test— The subject studies 20 full 
names, including first and last, and is required to write in 
the appropriate first name when the last names are presented 
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<n a different order. There is a total of 15 Items (three 
mins. for manorlzing, two mins. for testing). Score is the 
number correct (French, et a1., 1963). 

The Closure Flexibility factor has been defined as the ability to 
retain a complex Idea in spite of distraction. The marker tests are: 

Copying Test--Each it&n consists of a geometrical figure 
composed of four connecting line se^nents. The task is to 
copy the figure onto a square matrix of dots. There are 32 
figures (three mins.). Score is the number correct (French, 
et a1.. 1963). 

Closure Flexibility Test (Concealed Figures-Form A) — 
Each item consists of a figure on the left followed by a row of 
more complex drawings, some of which contain the original 
figure. The subject marks those drawings which contain the 
figure. Test developed by Thelma 6. Thurstone and T, E, Jeffrey. 
There are 40 probl&Ds (10 mins.). Score is the number correct 
minus the number incorrect. 

Designs Test— In this test of L. L. Thurstone 's (1938), 300 
designs are presented, in 40 of which the Greek capital letter 
"Sigma" is embedded. The task is to mark as many as possible 
of the figures containing the "sIgma" in a tfeK)-m1nute period. 
Score is the number correct. 

The Perceptual Speed factor has been described as the ability to compare 
visual configurations and identify two figures as similar or identical. 
The marker tests are: 

Finding A's Test— In each of several colianns of 41 words, 
the task is to draw a line through the five w)rds containing 
the letter "a". Score is the number of v«)rds correctly found 
in two minutes (French, et a1., 1963). 

Number Conparison Test— The subject examines pairs of 
multi-digit numbers and indicates whether the two numt^rs in 
each pair are the same or different. There are 48 pairs of 
items (1 1/2 mins.). Score is number correct minus the number 
Incorrect (French, et a1., 1963). 

Identical Pictures Test— For each it&n the subject is to 
check which of five numbered geometrical figures or pictures 
In a row is Identical to the reference figure at the left eiKJ 
of the row. There are 48 rows or items (1 1/2 mins.). Score 
is the number correct minus a fraction of the number incorrect 
(French, et a1 . , 1963). 
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The Syllogistic Reasoning factor has been described as the ability 
to reason from stated premises to their necessary conclusions. The three 
marker tests for this factor are: 

Nonsense Syllogisms Test— In this test, suggested by 
Thurstone's False Prenlses , the subject is presented with 
formal syllogisms made up of nonsense words so that they cannot 
be solved oy reference to past learning. The task is to 
indicate which of the stated conclusions follow logically 
frcwi the premises and which do not. There are 15 itens 
(4 mins.). The score is the number correct minus the number 
incorrect (French, et a1., 1963). A constant of 10 was 
later added to each subject's score to eliminate any negative 
numbers . 

Logical Reasoning Test— In this test developed by 
Guilford, the subject's task is to choose the correct 
conclusion that can be drawn from two given statements. 
The test is composed of 20 formal syllogisms, each with 
four response choices (10 mins.). Score is number correct 
minus a fraction of the number Incorrect (French, et al., 
1963). 

Inference Test— In this test adapted from Guilford, 
the subject's task Is to select the one of five conclusions 
that can be drawn from each given statement. There are 
10 Items (6 mins.). Score is the number correct minus a 
fraction of the number incorrect (French, et al., 1963). 

The Spatial Scanning factor has been defined as the ability to visually 
explore a wide or complicated spatial field. A planning ability may also 
be involved. The marker tests for this factor are: 

Maze Tracing Speed Test— The task is to find and mark 
an open path through a moderately complex series of paper 
mazes. There are 24 interconnecting mazes (3 mins,). Score 
is the number of mazes through which a line has been correctly 
drawn (French, et al., 1963). 

Choosing a Path Test— Each item of this test, adapted from 
AAF Printed Classification Tests (Guilford, et al., Eds., 1947), 
consists of a network of lines (as in an electrical -circuit 
diagram) having many intersecting and intermeshed wires with 
several sets of terminals. The task is to trace the lines 
and to determine for which pair of terminals, marked S (start) 
and F (finish), there is a complete circuit through a circle 
at the top. There are 16 items (7 mins.). Score is the numt^r 
of problems marked correct minus a fraction of the number 
incorrect (French, et al., 1963). 
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Map Planning Test— In this test, adapted fran AAF 
Printed Classification Tests (Guilford, et a1 . , Eds., 1947), 
the subject sees diagrammatic sections representing city 
maps. The streets are blocked at various points by barriers 
represented by circles. The task Is to find the shortest 
route between tM) given points w-*thout crossing any road- 
blocks. There are two maps with ten routes per map (3 mins.). 
Score is the numbsr correct (French, et a1., 1963). 

Three additional tests v^re administered. While not used frequently 
as marker tests, they were presumed to reflect various aspects of htanan 
information processing. These tests are: 

Grammatical Reasoning (A-B) Task— Each Item In this 
task, developed by Baddeley (1968) and adapted by Rose (1974), 
consists of a statement followed by a pair of letters 
(either AB or BA). The statanents claim to describe the 
order of the two letters (i.e., to say which precedes or 
follows the other). The subject's task is to determine 
whether each statement is a true or false description of 
the letter pair which follows it. The test is made up of 
two parts, each containing 32 itens (1 min./part). Score 
for each part is the number correct (Rose, 1974). 

Neisser Search Task—In this task, developed by Neisser 
(1967) and adapted by Rose (1974), the subject is given a page 
containing columns of groups of five letters and is asked 
to search for a particular letter or letters, placing a 
check next to each item (i.e., group) he finds with one of 
the targeted letter(s). There are six trials: in the first 
two trials the subject is given one letter to search for (20 
sees.); in the next two he searches simultaneously for two 
letters (30 sees.); and in the last tv», he searches simul- 
taneously for four letters (30 sees.). The second trial of 
each pair uses the same target(s), but the masking letters 
are different. Scores are: 1) the average time per correct 
itan (in sees.) over all conditions, and 2) the slope of the 
best fitting regression line of the time per itan (in sees.) 
by target set size function (Rose, 1974).* 

Permutations Task— In this task, developed by Leskow 
and Smock (1970) and adapted by Rose (1974), the subject 
is asked to write down on separate slips of paper (which 
he then turns face down) as many different license plate 
numbers containina only the digits 1, 2, 3, and 4 as he can 
think of (3 mins.). Scores are: 1) the total number of 



* The slope data, although computed, are not reported in the present 
study. 
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correct new permutations, and 2) a frequency count of 
nisnbers held constant from one trial to the next In the first 
position, and a similar count for the second position, given 
the first was leld constant.* 



* The frequency count data, although computed, are not reported in 
the present study. 

16 



RESULTS 



The results of three sets of analyses are presented below. The 
first set deals with the impact of criterion task manipulations on a 
variety of performance measures. The second set is concerned with the 
factor structure of the reference battery. The third and final set 
examines the relationship between ability factors and criterion task 
perfomwnce. 

Criterion Task 

The proportions of fault-finding problens solved under each of 
the nine (3 x 3) experimental conditions for each replication are 
presented in Table 2. A problem was defined as solved if the subject 
named the correct breakpoint within the 15 minutes allotted. From 
the proportions shown in Table 2, it is clear that the problems 
were generally soluble within the time limit provided. The most 
conspicuous cells are those representing the highest level of formal 
difficulty and percep tua l c ompjexity (3C). Approximately half of 
the subjects failed to solve this problem on their first encounter 
(Replication 1), but after continued exposure to the. test problens 
almost all subjects solved it during Replication 2. 

As part of a more detailed analysis of how the task manipu- 
lations affected various aspects of performance on the criterion task, 
a number of dependent measures were examined. Total time to solu- 
tion (maximum » 15 minutes) and number of trials to solution were the 
most basic* Both measures were tr&nsformed to normalize their 
distributions using natural logaritfsns. The geometric means collapsed 
across replications are presented in Figures 1 and 2 for time to 



For problems which were not solved, the maximum time score (15 
minutes) was assigned, and t:.c number of trials completed at that time 
wai used as the tibial score. 
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TABLE 2 



PROPORTIONS OF PROBLEMS SOLVED WITHIN 15 MINUTES (N«135) 





Repl ications 
















1 








2 




Formal Difficulty 


1 


2 


3 




1 


2 


3 


Perceptual Complexity A 


1.000 


.925 


1.000 


1 


.000 


1.000 


.985 


B 


1.000 


.993 


.926 


1 


.000 


1.000 


1.000 


C 


1.000 

L 


.993 


.542 


1 


.000 


l.(KK) 


.993 



Id 
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solution and number of trials to solution respectively. An analysis 
of variance performed on the time-to-solution data (see Appendix III) 
revealed significant main effects for difficulty (D), complexity (C), 
and replications (R). All interaction terms were also significant. 
The main effect of replication and its interactions with the other 
independent variables were, in general, significant for all of the 
dependent measures; however, the replication variable was of peripheral 
interest in the present report, and will not be considered further. 

Time to solution did increase as a function of both difficulty 
and complexity. A more detailed analysis of the D x C interaction 
showed that the differences between the A (lovrast) and C (highest) 
levels of perceptual complexity were significant at each level of 
formal difficulty (P-.Ol, Scheffe, 1959). Further, the size of the 
A-C difference (See Figure 1) varied significantly as a ^'unction of 
difficulty { p«r.001, Myers. 1972, p. 358), so that, while the 
effect of perceptual complexity is in general significant, its 
impact is largest on problems at the highest level of formal diffi- 
culty. 

The analysis of variance of trials-to-snlution data indicated 
a significant main effect of formal difficulty, but no significant main 
effect for perceptual complexity (see Appendix III). The 0 x C inter- 
action was significant; further, there was a significant interaction 
(see Figure 2) between A-C and difficulty (p-.OOl). The A-C 
difference was significant only at the highest level of difficulty 
(P' .01). Thus, the significant D x C interaction is accounted 
for by the fact thwt perceptual complexity impacts on trial s-to 
solution only for the highest level of formal difficulty. 



1 ' ^ 

Geometric mean ~ c n"' 



Z Ufs Xi) 

1=1 

N 



19 




Figure 1. Geometric mean time to solution as a function 
of formal difficulty and perceptual complexity. 
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Figure 2. Geometric mean trials to solution as a function 
of formal difficulty and perceptual complexity. 
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From these two analyses. It is clear that both independent 
variables impact upon criterion task performance, bmt their impact 
differs as a function of the dependent measure considered. In par- 
ticular, while COTiplexity strongly affects time to solution at all 
levels of formal difficulty, it only affects trials to solution on 
the JTOst difficult problems. 

Another dependent measure of interest is time per trial. For 
each problsn, the subject's time to solution was divid«l by his total 
number of trials, and the resulting scores were normalized using 
natural logaritNns. The geometric means collapsed across replica- 
tions are presented in Figure 3. An analysis of variance of the 
time-per- trial data showed all main effects and interactions to be 
significant (see Appendix III). The differences between the A and 
C levels of perceptual complexity were significant at all levels 
of formal difficulty (p .01) the A-C by difficulty interaction 
was also significant (p .005); perceptual complexity again had 
its greatest impact at the highest level of difficulty (see Figure 3). 

The final two dependent variables are measures of the efficiency 
of the fault-finding tests conducted by subjects. Efficiency was 
expressed in terms of the proportion of break{K)ints which were 
eliminated by each test a subject used relative to the maximum 
number which could have been eliminated on that test. Efficiency was 
scored for trial one (the first test) as well as averaged over trials. 
A more detailed discussion of the derivation of these measures is 
presented in Appendix IV. 

Mean trial-one efficiencies are presented in Figure 4 as a 
function of formal difficulty and perceptual complexity. An analysis 
of variance {see Appendix III) of these data showed a significant main 
effect for complexity, but not for difficulty. The D x C interaction 
was significant, so once again this interaction was analyzed in more 
detail. The A-C by difficulty interaction was significant (p<:.01). 
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Figure 3. Geometric mean time per trial as a function 
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Figure 4. Mean trial-one efficiency as a function of 
formal difficulty and perceptual complexity. 
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The A-C complexity contrast was significant at both the 1 and 3 levels 
of difficulty (p .01), but neither the A-C nor the B-C contrast was 
significant at the intermediate difficulty level. The interaction 
cannot be explained parsimoniously, but to speculate, it appears that 
the lA and IC values are spuriously far apart. Both are within the 
99% confidence interval around the IB value; thus, it may be that 
perceptual complexity impacts on trial-one efficiency primarily, once 
again, at the highest level of formal difficulty. 

An arcsin square root transformation was applied to the effi- 
ciency-per-trial data. The means are presented in Figure 5 as a 
function of perceptual complexity and formal difficulty. An analysis 
of variance (see Appendix HI) showed that the main effects of diffi- 
culty, and complexity, as well as the D x C interaction, were signifi- 
cant. The significant interaction appears once again to be due to a 
strong complexity effect at the highest level of difficulty. The 
A-C by difficulty interaction contrast was significant, and the A-C 
contrast was significant only at the highest level of difficulty, it 
seems from these final two analyses that perceptual complexity has 
its greatest impact on fault-finding efficiency when the formal 
difficulty of the problems is high. 

The analyses of the five dependent measures show that, while 
both difficulty and complexity affect the performance of subjects in 
the criterion task, they do not necessarily affect the saire aspects 
of performance, nor do they affect particular aspects in the same way. 
Table 3 presents "^^ ^o*" s^ch independent-dependent variable pair.* 



According to Fleiss (1969), u 2 -jg preferred 'statistic for 
ei-.timating the proportion of variance in Y accounted for by X. It 
wcs not convenient to estimate^--^2 in the present study; however, 
and u' 2 differ only in the extent to which the numerators and denominators 
are unbiased. Furthermore, the values for the two statistics differ 
only slightly when F values are high and when there Is a large number 
of independent observations of each experimental factor. Both these 
conditions were met in the present study. 
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Figure 5. Mean arcsin square root efficiency per trial as a 

function of formal difficulty and perceptual complexity. 
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TABLE 3 



RELATIVE SENSITIVITY OF DEPENDENT VARIABLES TO DIFFICULTY 
AND PERCEPTUAL COMPLEXITY IN TERMS OF ^ 2 







Dependent Measures 






Experimental 
Manipulations 


Time 


Trials Time/Trial Trial 


1 Efficiency 


Efficiency/Trial 


Formal 
Difficulty 


.82 


.79 .60 


.04 


.52 


Perceptual 
Complexity 


.70 


.03 .84 


.30 


.18 
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Time to solution and time per trial are influenced strongly by both 
difficulty and complexity, while triaUone efficiency is affected to 
a lesser degree by perceptual complexity, and almost not at all by 
difficulty. Trials to solution and. to a lesser extent, efficiency 
per trial are more sensitive to ctenges in difficulty than to changes 
in complexity. The only impact of perceptual complexity on trials 
to solution or efficiency per trial appears to occur at the highest 
level of formal difficulty. 

The interplay between task manipulations and performance 
measures is quite important, since the different measures may be re- 
lated to different (sets of) abilities. If this is tt^ case, the 
relationship between task characteristics and ability patterns may 
depend upon the performance measures urKier consideration. Thus, 
different regression equations might be necessary to predict differ- 
ent aspects of criterion performance from abilities, over and above 
any changes in predictive equations made necessary by manipulations 
of the criterion task itself. 

Reference Battery 

The intercorrelations among reference tests are presented in 
Table 4. Five major factors were extracted from this matrix using 
a principal components solution. Orthogonal rotation of the factors 
was performed using a varimax criterion. Table 5 presents the rotated 
factor loadings; the algebraic signs for Factor IV have been reflected 
for convenience. Factors were Interpreted for psychological meaning- 
fulness from the projections of the reference tests on the rotated 
axes. 

Factor I is defined in terms of the high loadings exhibited by 
six of the reference tests. Three of the tests. Copying, Closure 
Flexibility, and Designs have previously been used as marker tests for 
a Flexibility of Closure factor (see Table I above and French, at aU 
1963). Although French et al.,des1gnate the same three measures as the 
best deflners of a Flexibility of Closure factor, substantial loadings 
of other tests suggest that the obtained factor may be broader than 
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TABLE 5 



FACTOR LOADINGS* IN ROTATED FACTOR MATRIX 





Reference Tests 


Factors 
I II 


III 


IV 


V 


2 


1 




01 


66 


18 




1 w 




2 


Md7P TrAr^ino ^n^pd 


69 


-01 


-01 




no 


58 


«# • 


rnnvl no 


7 1 


08 








70 


4 


loolrdi RM^onlno 


27 


63 


18 




27 




5. 




18 


28 


IS 

1 9 




S2 


42 


6 

V • 




CO 


13 


no 




10 


4Q 


7 


Fir^t find t Act Nauiac 


-07 


26 


77 


10 






g 


1 ArA^innc 

i»WWO V 1 wild 


15 


05 


1 *? 


?7 




SO 


Q 


HK 1 Ap ^ •.MiitnKAv* 


10 


06 




10 


19 


77 


0 


Findino A*q 


33 


11 


12 

1 Cm 




•10 


3Q 


1 


^Mrrh fmuMn) 


-10 


-26 




-.77 


04 


fi7 


12 


NufTihpr CoitiOAricon 


05 


-11 




fi7 
w/ 




57 


3. 


Designs 


59 


-01 


29 


51 


04 


70 


14. 


Nonsense Syllogisms 


00 


78 


08 


00 


08 


62 


15. 


Closure Flexibility 


70 


24 


00 


37 


21 


72 


16. 


Choosing a Path 


74 


17 


05 


-20 


06 


63 


17. 


Inference 


05 


71 


02 


18 


06 


54 


18. 


Map Planning 


53 


11 


07 


38 


30 


54 


19. 


Permutations 


08 


21 


02 


-13 


69 


55 


20. 


Picture- Number 


07 


07 


83 




24 


77 


21. 


Figure Classification 


40 


61 


09 


01 


23 


60 



♦Factor loadings reflected and rounded to two places; decimals anitted. 



Factors are tentatively defined as: 

I - Flexibility of Closure/ Spatial Scanning 

II - Syllogistic Reasoning 

III - Associative Manory 

IV - Perceptual /Cognitive Speed 

V - Induction 



originally defined. The other tests with high loadings are Ma^e Trac- 
ing Speed, Choosing a Path, and Map Planning. These are the three marker 
tests originally cnosen for the proposed Spatial Scanning factor. Royce 
(1973), in his recent review of cognitive factors, has argued that 
both of these factors— Flexibility of Closure and Spatial Scanni ng--may 
be components of a second-order Visualization factor. AltN)ugh the 
evidence is incomplete, the first factor obtained in the present study 
Is tentatively interpreted as a combined Flexibility of Closure/Spa- 
tial Scanning factor. 

Factor II is defined primarily from high loadings of the Nonsense 
Syllogisms, Logical Reasoning, and Inference tests. These are the 
marker tests for a Syllogistic Reason1n^ ^ factor. Further support 
for this interpretation is the high loading of the Grammatical Reason- 
ing test, which clearly Involves "ability in formal reasoning from 
stated premises to rule out nonpermissible combinations and thus to 
arrive at necessary conclusions (Ekstrom, 1973)." 

Factor III is readily defined from high loadings of the Object- 
Number, Picture-Number, and First and Last Names tests as the Associa- 
tive Memory factor. This same factor Is defined by French, et aU using 
the same three tests, as the ability to remember unrelated bits of in- 
formation. 

Factor IV is defined by the high loadings of the Finding A's, 
Number Comparison, and Identical Pictures tests as the Perceptual Speed 
factor. French et al., defined this factor as "speed in finding figures, 
making comparisons, and carrying out other very simple tasks involving 
visual perception." In an attempt to add support to this interpreta- 
tion, an adaptation of one of Neisser's search tasks,, presunebly meas- 
uring some aspects of visual manory and perceptual speed, was included 
in the present battery. This test (Neisser Search Task) had a high 
loading on this factor.* 

It 

The measure of performance on this task was a time-par-iten measure; 
a lower score indicated better performance. This scoring rule explains 
why the test has a negative loading on Factor IV. 
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Factor V is defined primarily by the high loadings of the Letter 
Sets, Locations, and Permutations tests. French, et aU have designated 
the first two of these measures as two of the best definers of a cog- 
nitive factor known as Induction. It is defined as the ability to 
find and test out hypotheses which will explain sets of data. The 
same type of reasoning (as distinct from syllogiitic reasoning) was 
discussed by Leskow and Smock (1970) in terms of Piaget's stage of 
formal operations. Leskow and Smock devised a test which confirmed 
some of Piaget's postulations about this type of reasoning. This 
same test was adapted for use in the present battery (Permutations 
test) and as expected, loaded quite highly on this factor. 

Projection of Criterion Data on Reference Factors 

The purpose In conducting this final set of analyses vols to re- 
late the two basic criterion task characteristics to the pattern(s) of 
abilities contributing to performance under the various experimental 
conditions. While an analysis of the abilities related to performance 
under each of the (3x3x2) 18 conditions would have been of Interest, 
the most direct way of relating task dimensions to ability requirements 
was to focus on the marginals. Specifically, the marginals for formal 
difficulty (1, 2, 3, as difficulty increased) and perceptual compl exi ty 
(A, B, C, as complexity increased) were used. This approach was pre- 
ferred to that of using the 18 Interactive marginals since the latter 
could not be ordered a priori. Without such ordering, interpretation 
of the pattern of loadings on the ability structure would be difficult 
at best. 

A Stoloff regression procprdnre (Stoloff, 1973) was performed to 
obtain coefficients representing the estimated loadings of the criterion 
measures on the factor structure underlying the reference battery. Table 
6 presents the intercorrelations of the ability reference tests with 
the five performance measurc;s for each level of formal difficulty and 
perceptual complexity. The coefficients estimated by the Stoloff pro- 
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cedure are presented in Table 7. The communal i ties (h^) in Table 7 
show that very little of the time oer trial vai iation was accounted 
for by the reference fdctors (h2=.08, N = 135, p--.01); thus, 
this measure was excluded from further analysis. From 10% to 30% 
of the variation in the other dependent measures was accounted for. 
Further, this common variance was primarily associated with Factors 
I, II, III, and V; all but two of the Factor IV coefficients were 
approximately zero. Generally small correlations were also found 

between the reference tests defining Factor IV and the criterion 
conditions (as shown in Table 6). 

The relationships between criterion performance and abilities are 
clarified in Figures 6 through 9. These figures show the changes in 
loadings as a function of levels of formal difficulty and perceptual 
complexity , for each performance measure except time per trial. In 
each plot, any factor which had at least one coefficient greater than 
.20 was included. 

Certain patterns are apparent from these figures. First, the 
loadings on Factor I (Flexibility of Closure/Spatial Scanning) gener- 
ally tended to increase as both formal difficulty and perceptual com - 
plexity increased. This relationship held across all four performance 
measures. Second, Factor II (Syllogistic Reasoning) was moderately in- 
volved in all but the time to solution measure. Further, its level 
of involvement was fairly constant, i.e., it did not change as a function 
of formal difficulty or perceptual complexity .* Next, the loadings 
on Factor V (Induction) tended to decrease as a function of increasing 
perceptual complexity on the trials to solution and efficiency per 
trial measures. On the remaining measures, loadings for this same 
factor were insignificant, or moderate but constant. Finally, Factor 
III (Associative Memory) appeared in only two plots, where its loadings 



* Recall that the confidence intervals for regression coefficients 
are quite broad as the coefficients approach zero (cf. Hays, 1963, 
p. 522). Thus, the notion of "constant" is appropriate, even though 
not apparent in the Figures. 
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TABLE 7 

ESTIMATED FACTOR LOADINGS* OF CRITERION VARIABLES ON REFERENCE FACTOR STRUCTURE 









Factors 












Criterion Measures 




I 


II 


III 


IV 


V 




R 


Trial 1 Efficiency 


1 


27 


18 


16 


09 


10 


15 


38 




2 


26 


21 


05 


06 


10 


13 


35 




3 


33 


19 


21 


03 


07 


19 


44 




A 


19 


on 
20 


13 


01 


17 


12 


35 




B 


32 


25 


13 


03 


09 


19 


43 




C 


35 


15 


17 






^ ft 

19 


A A 

44 


Ln Trials to Solution 


1 


15 


20 


09 


-09 


17 


10 


33 




2 


30 


22 


09 


-08 


15 


17 


42 




3 


34 


12 


04 


-00 


10 


14 


38 




A 


19 


19 


12 


-08 


30 


19 


43 




6 


27 


17 


04 


-13 


08 


13 


36 




C 


34 


20 


07 


03 


07 


17 


41 


Arcsin Square Root 


1 


30 


30 


20 


-04 


12 


23 


48 


Efficiency per Trial 


2 


33 


35 


19 


-02 


23 


32 


57 


3 


OU 


31 


12 


01 


14 


38 


62 




A 


34 


31 


30 


01 


28 


38 


61 




B 


38 


35 


IS 




11 


3 i 


DD 




C 


42 


31 


05 


-02 


14 


30 


54 


In Time to Solution 


1 


25 


14 


05 


-06 


29 


18 


42 




2 


25 


14 


-04 


-04 


23 


13 


37 




3 


32 


10 


-08 


-04 


29 


21 


45 




A 




1 i 


-04 


-05 


31 


15 


38 




B 


27 


13 


-01 


-09 


23 


15 


38 




r 
li 


41 


17 




no 




CO 


DO 


Ln Time per Trial 


1 


16 


-01 


-02 


00 


18 


06 


24 


2 


10 


03 


-09 


00 


17 


05 


22 




3 


08 


02 


-n 


-04 


21 


07 


26 




A 


10 


01 


-12 


-01 


17 


05 


23 




B 


10 


03 


-03 


-01 


18 


05 


21 




C 


13 


01 


-09 


-02 


22 


07 


27 



Signs have been reflected to relate superior perfomance to 
superior ability; decimals omitted. 



Factors are tentatively defined as: 

I - Flexibility of Closure/Spatial Scanning 
II - Syllogistic Reasoning 
III - Associative Memory 
IV - Perceptual/Cognitive Speed 
V - Induction 
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decreased as a function of perceptual complexity for the efficiency-per- 
trial measure and were small and constant as a function of formal diffi- 
culty on the trial -one-efficiency measure. 



42 



DISCUSSION 



The two tdsk manipulations, formal difficulty and perceptual cm- 
plexlty, strongly and systematically affected performance. The effect 
was in the expected direction; as formal difficulty and perceptual 
complexity increased, performance deteriorated. However, various aspects 
of performance were affected differentially by the task manipulations. 
This is reflected by the relative sensitivity of the dependent variables 
in terms of ^ ^ (Table 3 above). Trials to solution and time to solution 
were both influenced to a large degree by formal difficulty. This effect 
was almost "forced" by the problem construction; an ideal troubleshooter 
would have to use more trials (and consequently more time), since he 
had to test more breakpoints as formal difficulty increased. Perceptual 
complexity, on the other hand, would be expected to influence time to 
solution, and to have little effect on number of trials. This expectation 
stems frcHTi the topological equivalence of the problans. To the extent 
that subjects noticed this equivalence, perceptual complexity should have 
no effect on the number of trials, since each test is logically independent 
of the circuit configuration. However, time to soiution should increase, 
since the subject must sort c. *" test paths from increasingly confusing 
circuit diagrams. In fact, time to solution did increase across levels of 
perceptual complexity, while trials to solution did not, except at the 
highest level of formal difficulty. This last finding probably indicates 
that topological equivalence is harder to detect in the most difficult 
problems . 

The resu'fts for the other measures are not so easily interpreted, 
mainly because assumptions concerning subjects' strategies or informa- 
tion processing skills are necessary. For example, trial 1 efficiency 
is affected considerably more by perceptual complexity than by formal 
difficulty. It could be hypothesized that subjects develop some low- 
level tactic for deciding what the first test should be (e.g., "look 
for the longest circuit of OR gates", or more simply, "test the top- 
most terminal path"). These tactics could be iraintained across the 
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formal difficulty dimension; however, they might be more difficult to 
implement for the more perceptually confusing problems, when, for example, 
the longest circuit of "OR" gates is difficult to locate. Similarly, 
any other strategy making use of the conf igurational properties of the 
diagrams would probably be altered when the perceptual complexity is 
changed. 

On the other hand, efficiency per trial is affected more by formal 
difficulty than by perceptual complexity. This finding could be accounted 
for by assuming that subjects were choosing a strategy which did not 
depend on the conf igurational properties of the problems but which was 
sensitive to changes in formal difficulty. Such would be the case, for 
example, if subjects chose a strategy of testing a constant number of 
breakpoints. 

The time-per-trial measure is particularly difficult to interpret. 
This may be due to the inherent confounding it represents between trials 
(sensitive to difficulty) and time (sensitive to both difficulty and 
complexity). 

While there are no straightforward ways of confinning or dis- 
confirming these "strategy" hypotheses in the present experiment, there 
is another way of viewing these results. It is possible to examine 
the abilities contributing to performance on each dependent measure 
under the different task conditions. Hopefully, ability requirements 
can be formulated for various task denands. Such a formulation has 
direct bearing on several types of personnel decisions, especially 
those involving training and selection. 

One general question implied by a consideration of personnel 
selection problems is the following: given the ability requirements 
for one form of a criterion task, what will the requirements be if 
the task is altered in some way? In the present context, this question 
can be r-eformulated to ask several particular questions. Are the same 
abilities Involved as a task is varied across a single dimension, or 
are other abilities "brought into play"? For example, are the abilities 
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which covary with perfomiance on the 4-gate problems (fonital diffi- 
culty, level 1) the sai:;o abilities that covary with performance on the 
6«gdte problems (formul dittkulty, level 3)? Siniildrly, can abilities 
predict between dimensions (e.g., are the abilities which covary with 
formal difficulty the same as those which covary with perceptual com- 
plexity)? Furthermore, given the state of affairs discussed above, 
ndi.iely that different aspects of perforniance are measured by the diff- 
erent dependent variables, a sin<i1ar set of questions could be posed 
for the different measutes. Can a single set cf abilities predict 
several different performance measures? For example, do the abili- 
ties which covary with trial 1 efficiency also covary with total time 
to solution? 

Answers to these questions have both theoretical and practical 
importance. From a theoretical perspective, the answers might give 
a clearer indication of the strategies subjects used than could be 
inferred from the dependent measures alone. This might enable a 
better understanding of the kinds of task variations or performance 
demands that would produce ctianqes in ability requirements. Practically, 
answers would provide inforntdtion as to the reldtionship betv^en 
task characteristics and ability requirements for use in personnel 
decision procedures. Ideally, one would "ike the following kind of 
statements: "Given that task A will be performed under conditions 1, 
2, and 3, personnel selection (or training) would be different for 
each condition. Furthermore, people with ability profile X will 
perform well on 1 and 2, but not 3, etc." 

The results presented in Table 7 and Figures 6 through 9 deal 
directly with the above questions. On a conceptual "Jevel, there are 
several possible patterns of factor loadings that could have resulted. 
Of interest here are: 1) abilities that have a relatively constant 
pattern of loadings across both task dimensions; 2) abilities whose 
loadings increase or decrease consistently within one dimension 
but not the other; and 3) abilities that "appear" for only certain 
of the dependent measures. In fact, examples of such patterns did 
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emerge. Loadings on Fdvior I (Flexibility of Closure/Spatial Scanning) 
consistently increased as perceptual complexity increased; this held 
for all of the dependent measures. Likewise, as formal difficulty 
increased, the loadings on Factor I tended to increase. The loadings 
on Factor II (Syllogistic Reasoning) in general remained constant 
across all measures (except total time) and both task dimensions. 
Factor III (Memory) showed decreasing loadings as formal difficulty 
and complexity increased on the eff iciency-per-trial measure, but 
generally the reverse on trial 1 efficiency, and only small loadings 
on the other measures. Factor IV (Perceptual Speed) was unrelated 
to any of the measures of criterion performance. Factor V (Induction) 
showed decreasing loadings on the trials-to-solution and trial-1- 
efficiency measures, as both difficulty and complexity increased, 
and reasonably high loadings with no consistent patterns on the other 
dependent measures. 

Thus, it can be concluded that different abilities are involved, 
and at different levels of involvement, when either the task dimen- 
sions vary or different dependent measures are examined. If one were 
to use ability criteria to pre-select who would do well on these prob- 
lems, the choices would differ depending on the task characteristics 
and the criterion measure selected. As an illustration of possible 
personnel decisions, consider, for example. Figure 8, the loadings 
for the efficiency-per-trial measure. If we were to select a group 
to solve "A" configuration probl^s efficiently, independent of formal 
difficulty or speed, we would pick those subjects who had fairly high 
scores on Factors I, II, III and V. Furthermore, each of these abilities 
would be given approximately equal importance (in othe>- words, the cutoff 
value would be comparable across each of the factors). Now suppose 
we had a suspicion that the task might be performed under more perceptually 
conplex circumstances (the "C" configuration). In that case, we would 
select subjects with high scores on Factor I and II while the requirements 
for good scores on Factors III and V v«)uld no longer be appropriate. 
Further, Factors I and II would not receive equal weight in the decision; 
Factor I would be weighted more heavily. If another criterion of 
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performance were considered desirable, for example, total time to solu 
tion (Figure 9), different standards would be necessary. 

There are two general types of training or selection decisions 
implied by the data. One type of decision, typified by the pattern 
of loadings on Factor I, occurs when a change in task demand (due 
either to different criterion measures or changes witMn a task 
dimension) does not involve different abilities. Rather, a change in 
cut-off values is the appropriate decision. The abilities involved 
are the same, but are involved to a greater or lesser degree as the 
task demand changes. The second type of decision occurs when the 
task danand is altered and when a different set of abilities are 
involved, as occurred in the efficiency example (Figure 8). In this 
type of situation, since different abilities (or factors) are rele- 
vant, the implication is that different subjects should be selected. 

It v«)uld be desirable if one kind of task manipulation (or one 
aspect of performance) consistently implied one or the other type 
of decision. There was no consistent relationship between type of 
manipulation (or measure) and pattern of ability requirement in the 
present study. It may be that such consistent relationships could be 
explored when more control is exercised over the possible strategies 
subjects might utilize. The problem is that any change in a given 
task characteristic might be perceived as a new problem by sane 
subjects, but as the same problem by others. To the extent that 
these perceptions lead the first group to adapt alternative strate- 
gies for the "new" problems, different abilities may be Involved. 
The second group, while maintaining their strategy, would utilize 
the same abilities, but perhaps at different levels of involvement. 
Hopefully, further investigations, using the experimental -correla- 
tional approach ©nployed in the present study, will shed further 
light on these Issues. 
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This appendix contains black and white renditions of the actual 
circuit diagrams used in the experiment. Formal difficulty is coded by 
numbers, and perceptu a l complexity by letters, in order of increasing 
difficulty or complexity. Thus, problem 3C is the most difficult and 
most complex, while IC is the least difficult and most complex. In the 
actual diagrams of problems involving the B and C levels of perceptual 
complexity , wire secpnents were represented in randomly chosen colors 
(blue, green, red, black) to further influence perceptual complexity . 
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Welcome to the American Institutes for Research. I am , 

and win be working with you during the course of this experiment. Before 
we get down to work this morning, I'd like to take care of a few adminis- 
trative details and then give you a brief overview of what it is you'll 
be doing today. First of all, I need to know your name; then I'd like 
you to tell me whether or not you are color blind or have had experience 
with computers or symbolic logic. Could you also give me your age, the 
title of the last course you've taken in mathematics (note whether course 
was taken in high school or college), and your college major. 

As you know the experiment in which you are taking part today is 
concerned with probl©n solving— specifically with how v^ll you can locate 
breakpoints (or points of malfunctioning) in problems involving electrical 
circuits. In the real world, people who perform such tasks are called 
troubleshooters. The data which you help us generate will aid in answering 
several research questions. For instance, if sme people are better at 
troubleshooting than others, we would like to find out why in terms of 
individuals' basic aptitudes or abilities. We would also like to find out 
what happens to performance as variations in the task are introduced, making 
the task easier or more difficult. 

In a few minutes we will begin training you for the troubleshooting 
task. First we will acquaint you with some of the basic concepts and 
terminology which efficient troubleshooters use, and then familiarize you 
with the techniques of troubleshooting. Following the training session, 
you will be presented with a set of troubleshooting problems to solve. We 
will then take a rest break. After the break you will be taking 21 paper- 
and-pencil tests covering many aspects of human ability. These tests were 
selected because they are either known or hypothesized to be related to 
performance on the problem-solving task. We think you'll find this test 
battery to be both interesting and fun. "Pass" and "Fail" have no meaning 
on these tests, so relax and enjoy them, and simply try to do your best. 
We anticipate finishing by about 3:30 p.m. You will be paid for your 
participation at that time. 
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We are now ready to begin. Any questions at this point? Please 
stop me at any time in the Instructions if something is not clear. 

Now, if you will please give me your attention at the front of the 
room {£ stands by easel where a variety of circuit diagrams are presented). 

In the real world, vm often make decisions based on certain conditions 
which must be met. For example, if a driver sees either a red light OR a 
stop sign OR an obstacle, he must stop. This situation can also 
be depicted symbolically as an electrical diagram (Picture 1). The light 
in this picture (point) will go on if either swiltfi 1 "or" switch 2 is 
depressed, since electrical current will flow frm either switch on the 
left through the OR box and enable the light on the right to light. Deci- 
sions can also Involve a number of "AND" conditions— for instance, if a 
driver arrives at an intersection when the traffic light 1s green AND the 
path is clear, the driver can then go through the Intersection (Picture 2). 
Here, the light will only go on if— and only if--both switch 1 and switch 
2 are depressed since current must flow from both wires on the left 
through the AND box in order for the light on the right to light. It is 
Important that you understand how these AND - OR conditions operate in 
order to solve the troubleshooting problems which you will be working 
on a little later. 

We will now go into the mechanics of what you will be doing. Please 
open your notebooks to Example 1. Before you is a simple electrical 
diagram with two OR boxes. By depressing any one of the numbered switches 
on the left, you initiate current through the wires. The circles which 
are labeled A - F represent light sockets. The numbered stars indicate 
points at which breaks in the circuit [uax occur. In each problem there 
will be one , and only one, point where a break exists— all other points 
are in good working order. Your task will be to locate the one point which 
is not v^rking. 

Now, let's assume for a moment that all points in the system are 
wrking, and you have an imaginary light bulb. How would you get light 
A to go on? B? C? D? E? and F? Notice you cannot light "D" by 
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depressing switch 3. The reason for this is that current cannot flow 
backwards. Notice the arrows Inside the OR boxes; they will tell you the 
direction of current going out of the boxes. All other wires caning in 
contact with the box are input wires. 

Now let's look at the consequences of there being a break at some 
point in the circuit. Looking at the diagram before you, if the break 
point were at *4, and you put a lightbulb in socket F and depressed switch 
3, would the light go on? (Pause.) The reason it would not go on is 
because current cannot continue to flow through a wire beyond the break 
point. In this example, therefore, current cannot get through the OR 
box to light the light. In the same diagram, if there were a break at *5, 
could "E" light by depressing switch 1? (Yes.) How about "F"? (No.) 

Now turn to Example 2. Here we encounter one OR box and one AND box. 
What would you do to light "F"? (#1, #3F or #2, #3F). Do you see that 
if you put a light at F, and simply depress switch 1, F could not light? 
(The AND box needs two inputs.) Remember, your task during testing will 
be to discover at which of the possible break points in each problem 
a break actually occurs. In order to locate this breakpoint you will be 
checking different pathways by inserting a light bulb in one of the 
indicated sockets and depressing the proper switch or switches in order 
to turn the light on. If a light in a circuit doesn't work, you would 
know that the breakpoint is somewhere in that circuit and in no others; 
on the other hand, if a bulb lights, all points tested along that pathway 
must be working. 

For example, if this were a real test problem, you v«>uld know that 
one, and only one, out of the 4 points was broken. Your task is to 
find out which one. Let's say you start by putting a bulb at E and pressing 
switches 1 and 3. If E lights, which points would you know are good and 
can be eliminated? (No break at *1 or *3,) However, if E doesn't light, 
you would know the break must be at *1 or *3. Notice that you gain 
information if the light lights or if it doesn't light. Now, what do 
you do from here? Let's assume now that E doesn't light, and that we 
know the break is at either *1 or *3. To determine at which of the two 
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points the break exists, we could try putting our bulb at C and depressing 
switch 1. If C lights we would know *1 was not the break. Therefore 
it must be at *3. However, if C did not light, the break would be at *1 . 

Turn to Example 3. The problems you will be doing will vary in 
complexity. All problems, however, will still involve only "or" - "and" 
conditions and will contain only one breakpoint. Your task will be to 
discover the breakpoint in the most efficient manner possible. This means 
that it is important for you to work as quickly as possible and perform 
the least number of tests possible. You will be scored in comparison with 
an optimal troubleshooter whose tests always gain him the maximum amount 

of information Looking now at the third sheet, how would you light 

6? (1, 2G; 3, 4G). If there were a break at one of the five points in 
this problem, how would you begin to locate this breakpoint? (Which points 
can be eliminated? Which points are suspicious now?) 

Turn to your fourth example. How would you turn on the light at 6? 
(1, 3G or 1, 4G; 2, 3G or 2, 4G) How would you go about locating a break 
in this probl«Ti? (Points eliminated? Points suspect?) 

Turn now to the fifth example. In this training problem, we will 
consider some special properties of the boxes. Notice an OR box has two 
outputs. An AND box may also have tv^ outputs. If you press switch 1, 
assuming all points are in working order, could you light D, E, F, and 
what about H? (Yes.) Why? The OR box has two separate outputs which 
both send current through the AND box. Switch 2 will light H in the 
same manner. However, switch 3 alone will not light 6 or H. Let's 
consider what would happen If we depressed switches 1 and 2 and put a 
lightbulb at D. If there were a break at *1 or *2, would D light? (Yes.) 
Why? If I told you the light doesn't go on, where does the break have 
to be? So by depressing both switches 1 and 2, no information Is gained 
concerning *1 or *2. Similarly, if you were to depress switches 1 and 3 
and place a bulb at H, you would receive no information about points *4, 
*5, *6 since current can travel through the bottom OR box via the circuit 
from switch 1 or fran switch 3. What would your first test be in this 
problem to locate the break? (Points eliminated? Points suspect?) 
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Turn to Example 6, your last training problem. This problem Is 
slftHar to those you will see during testing. How would you light J? 
(1, 3, 4J or 2. 3, 4J.) H. would you light K? (IK. 2K, 3, 4K.) Let's 
find the break In this problem. How would you begin? (Points eliminated? 
Points suspect?) Any questions or comments? 

We are now re^dy to begin the testing session. There are a total of 
23 test problems. Each problem has only one breakpoint and It Is your 
task to discover this breakpoint. In order to locate It, you will perform 
tents one at a time in the same manner we have been doing. You will 
write down each test on a card indicating which switch (or switches) you 
wish to depress and the letter of the light you wish to test. (For 
exiiniple, in the last training problem, if you wished to put a light at 
u and press 1, 3, & 4, we want you to write "1, 3, 4J" on your card. 
Or if you wished to test for a break at *1 or *4, you would write IG 
on your card.) Ple?:>e write as legibly as possible. Give this card 
to the experimente who will inforr. you if the light will light (your 
green "yes" light will light) or if the light will not light (your red 
"no" light will go on.) If you have made an erroneous test— such as a 
test for which there is no pathway connecting the switch and light indi- 
cated on your card— .he yellow "error" light will go on. For example, 
in the last training problem, if you depressed switch 1 and put a light 

^» ^ could not light. (Do you see what's wrong?) Another error 
would occur if you did not have sufficient current through an AND box 
to light the light— for example, if you depressed switch 3 and put a 
light at E, E would not light. 

You will continue to write test cards and give them to the experimenter 
until you have found the breakpoint. Make tests until you know where i,he 
breakpoint is. Then write the breakpoint number on a card, and the 
experimenter will confirm the correct answer. Remember you will be scored 
in terms of speed (how fast you perform) and efficiency (how much information 
you gain 1)> f «• ^est). Please do not guess where the breakpoint lies. 
?s this will lo'.er your efficiency considerably. 
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Ts everyone ready to begin? Feel free to smoke during the session 
if you wish. Please do not make any notations in your book about the 
tests you have made. During breaks, do not discuss these problems among 
yourselves. 
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TABLE III-l 

ANALYSIS OF VARIANCE OF CRITCRION DATA 
(Ln Time to Solution) 





Of 


MS 


F 


S (Subjects) 


134 


1.3:^4 




C (Complexity) 


2 


50.064 


318.364** 


D (Difficulty) 


2 


139.940 


632.738** 


R (Replications) 


1 


90.956 


356.306** 


SC 


268 


.157 




SO 


268 


.221 




SR 


134 


.255 




CD 


4 


8.499 


52.607** 


RD 


2 


2.884 


16.507** 


RC 


2 


3.656 


18.694** 


SCO 


536 


.162 




SRD 


268 


.175 




SRC 


268 


.196 




RCD 


4 


.806 


3.699* 


SRCO 


536 


.218 





* p .01 
** p .001 
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ANALYSIS OF VARIANCE OF CRITERION DATA 
(Ln Trials to Solution) 



Source 


df 


• 


F 


S (Subjects) 


134 


.398 




C (Complexity) 


2 


.470 


4.390 


D (Difficulty) 


2 


73.828 


503.950** 


R (Replications) 


1 


6.912 


55.710** 


SC 


268 


.107 




SD 


268 


.147 




SR • 


134 


.124 




CD • . . 


4 


.480 


4.709* 


RD 


2 


1.703 


14.070** 


RC 




1.24C 


9.327** 


SCD 


536 


.102 




SRD 


268 


.121 




SRC 


268 


.134 




RCD 


4 


2.037 


14.353** 


SRCD 


536 


.142 





* p .01 
** p .001 
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TABLE 111-3 

ANALYSIS OF VARIANCE OF CRITERION DATA 
(Ln Time Per Trial) 



Source 


df 


MS 


F 


S (Subjects) 


134 


• 1.432 




C (Complexity) 


2 


41.017 


683.078** 


D (Difficulty) 


2 


15.543 


198.656** 


R (Replications) 


1 


47.690 


323.130** 


SC 


268 


.600 




SD 


268 


.782 




SR 


134 


.148 




CD 


4 


5.149 


88.484** 


RD 


2 


2.062 


37.600** 


RC 


2 


2.463 


47.054** 


SCO 


536 


.582 




SRD 


268 


.548 




SRC 


268 


.523 




RCD 


4 


2.227 


35.626** 


SRCD 


536 


.625 





** p .001 # 
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TABLE III -4 

ANALYSIS OF VARIANCE OF CRITERION DATA 
(TRIAL 1 EFFICIENCY) 



Source 




MS 


F 


S (Subjects) 


]j4 


.324 




C (Complexity) 


2 


4.402 


56.946*** 


D (Difficulty) 


2 


.176 


1.865 


H (Replications) 


1 


1.386 


20.927*** 


SC 


268 


.773 




SD 


268 


.946 




SR 


134 


.662 

• W x/ Cot 




CD 




.931 


12.678*** 


RD 


2 


.158 


2.377 


RC 


2 


.542 


.979 


SCO 


536 


.734 




SRH 


26a 


.666 




SRC 


268 


.554 




RCD 


4 


.115 


1.662 


SkCD 


536 


.693 





*** p .OU05 
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TABLE 1 1 1-5 

ANALYSIS OF VARIANCE OF CRITERION DATA 
(Arcsin Square Root Efficiency Per Trial) 



Source 


df 


MS 


£ 


S (Subjects) 


134 


.337 




C (Complexity) 


2 


1.400 


29.450** 


D (Difficulty) 


2 


8.260 


144.909** 


R (Replications) 


1 


1.034 


21 .961** 


SC 


268 


.475 




SD 


268 


.570 




SR 


134 


.471 




CD 


4 


.407 


10.216** 


RD 


2 


.294 


5.941* 


RC 


2 


1.385 


27.374** 


SCO 


536 


.399 




SRD 


268 


.495 




SRC 


268 


.506 




RCD 


4 


.598 


11.125** 


SRCD 


536 


.537 





* p .01 
** p .001 
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APPENDIX IV 
DERIVATION OF EFFICIENCY MEASURES 
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Efficiency was defined as the number of breakpoints eliminated by 
a test divided by the number of breakpoints which could be eliminated by 
the optimum test on that trial. For each trial, the scorer determined 
the breakpoints which might still be faulty (i.e., about which the subject 
could have had no definite information). The subject's test eliminated 
some number of these breakpoints; this number was less than or equal to 
the number of breakpoints which could be eliminated by the best available 
test. The efficiency for that trial was the ratio of these two numbers. 

For example, on the first trial of the simplest problem (lA), there 

are nine breakpoints, exactly one of which is known to be faulty. The 

optimal first test in this situation would eliminate four breakpoints from 

further consideration (if the probe light lights), or five breakpoints 

(if the probe light falls to light). If the probe light fails to light, 

a test is available for two of the five remaining breakpoints, and so on, 

following an appropriate split-half technique. Thus, on trial one, 

the optimal test eliminates either 4/9 or 5/9 of the breakpoints, and the 

optimal score Is defined as the value less than or equal to 1/2 (in this 

case 4/9). A subject might perform a trial-one test which eliminates n 

breakpoints (if the light lights), or 9 - n breakpoints (if th': light 

doesn't light). The subject's efficiency value is n/9 or 9 - n/9, 

whichfcver is less than or*equal to 1/2. His trial-one efficiency score 

is defined as his efficiency value divided by the optimal score, e. g., 
n /4 9 - n /4 

1^/^ or — ^ — /-^ . On scoring trial two, the scorer determined the break- 
points which were still possibly faulty, as a result of the subject's pre- 
ceding test. The optimal test for these remaining breakpoints was then 
determined and the optimal score was calculated. On this and subsequent 
trials the obtained efficiency value was the nidnber of remaining break- 
points eliminated by the subject's test, divided by the nimiber of possibly 
faulty breakpoints before the test; the efficiency score is the efficiency 
value divided by the optimal score. Thus, at each trial, the nimiber of 
breakpoints which are logically tenable are used to examine a subject's 
perfoniiance. He deviates from perfect scores to the extent that he 
retests breakpoints already logically eliminated, or fails to find the 
most efficient rest based on all logically available Information. 
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Two hypothetical protocols are presented in Table IV-1. Assume 
an eight-breakpoint problem, where split-half or nearly split-half tests 
are always available. A represents a potential breakpoint which has 
not yet been shown to be either faulty or non-faulty, a is a breakpoint 
logically shown to be non-faulty, the underlined breakpoints are those 
under test, and "1" indicates the light lit, while "0" indicates the bulb 
failed to light. Subject #1 is an ideal troubleshooter, while #2 is not. 
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